coding and cooking

Creating and configuring AWS Elasticsearch using v4 signature in a Rails application

It's easy to setup a Rails app to access an AWS Elasticsearch instance through version 4 signature. This are the versions of the resources used:

  • Ruby version 2.1.8
  • Rails version 3.2.22
  • gem elasticsearch-transport-aws4 version 2.0.0
  • gem faraday_middleware-aws-signers-v4 version 0.1.6

Creating the AWS Elasticsearch instance

It's very easy to create and configurate your AWS Elasticsearch instance. Just follow the steps below.

Enter your AWS Console and search for Elasticsearch

Following the image sequence below, search for Elasticsearch service on AWS:

  • Click in Services (the first menu button)

  • Enter 'elasticsearch' on the input and choose the 'Elasticsearch Service'

  • On Elasticsearch Service dashboard, click the button Create a new domain

Choose the instance configuration. Any doubts, just click in the question mark on right top - same line of the title, 'Configure cluster'.

  • Define the policy

Click in Modify access policy when changing or select a template on the dropbox on the view Set up access policy, the next view on the instance creation process:

Select the template Allow or deny access to one or more AWS accounts or IAM users and, in the popup that will appears, put your Account ID - usually a 12 characters number. Here, to our example, 012345678901. You'll have something like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "012345678901:root"
        ]
      },
      "Action": [
        "es:*"
      ],
      "Resource": "arn:aws:es:us-east-1:012345678901:domain/instance_name/*"
    }
  ]
}


Now you have your Elasticsearch Instance working and waiting for requests. But still there is some other fiew steps to go. We need to ensure to our account access to Elasticsearch resources in our console. Just keep going...

Setting AWS account 'free pass' to your Elasticsearch instances

Now you need to give to your account the credentials to access Amazon Elasticsearch.

Accessing your user settings

Click in your name/account ID as shown in the image below:

Now, click in My Security Credentials:



Creating a group and adding your user to it

First, click in Groups to create a group:

Then, select the group you created and add your user, clicking on the blue button Add Users to Group:

On the next screen, just select the users and click the right bottom blue button Add Users.

Attaching the Elasticsearch Policies to this group

On the screen right before choosing the users to add to group - the group summary screen - click on the tab Permissions:

Then click on the blue button Attach Policy. On the list will open, search for AmazonESFullAccess policy: 

Check on the checkbox on the left and click the blue button on the right bottom of the page Attach Policy. It's done! 
Now our AWS account has total access to any of our Elasticsearch instances. \o/

Configuring Rails app to access AWS Elasticsearch Instance

Now, to configure Rails to access the instance, you need to create an Elasticsearch configuration file in initializers

config/initializers/elasticsearch.rb

Now, there you will set the connection stuff:

require 'faraday_middleware/aws_signers_v4'

options = if Rails.env == 'development' || Rails.env == 'test'
            { url: ENV['ELASTICSEARCH_URL'] }
          else
            transport_configuration = lambda do |f|
              f.request :aws_signers_v4,
                credentials: Aws::Credentials.new(ENV['AWS_ACCESS_KEY'], ENV['AWS_SECRET_KEY']),
                service_name: 'es',
                region: ENV['AWS_REGION']

              f.response :logger
              f.adapter Faraday.default_adapter
            end

            transport = Elasticsearch::Transport::Transport::HTTP::Faraday.new hosts: [
              {
                scheme: 'https',
                host: ENV['ELASTICSEARCH_URL'],
                port: ''
              }
            ], &transport_configuration

            { transport: transport }
          end

Elasticsearch::Model.client = Elasticsearch::Client.new(options)


where:

  * ENV['ELASTICSEARCH_URL'] is the Elasticsearch instance URL to make your app uses it in your machine. For example, the local ELASTICSEARCH_URL could be someting http://localhost:9200 or any other port your Elasticsearch is running in your machine.
  * ENV['AWS_ACCESS_KEY'] is, as it says, your AWS ACCESS KEY to your account.
  * ENV['AWS_SECRET_KEY'] is, as it says, your AWS SECRET KEY to your account.
  * ENV['AWS_REGION'] is the region your Elasticsearch is running on AWS.

Depends on you how do you define your environment variables in your application. I like to use the gem dotenv-rails for this; that allows me to define my env variables in a .env file. So, if you use dotenv-rails, for example, your local env variables file, to use a local Elasticsearch instance, could be something like this:

ELASTICSEARCH_URL=http://localhost:9200

And the .env file in your app server, where is the app that access the AWS Elasticsearch instance, could be something like this:

  * ELASTICSEARCH_URL=url_to_your_elasticsearch_service -> something like search-instance-xireaksn64i3md8h2n0fkd.us-east-1.es.amazonaws.com, for example
  * AWS_ACCESS_KEY=your_user_aws_access_key -> something like ABCDEFGHIJKLMNOPQRST, for example
  * AWS_SECRET_KEY=your_user_aws_secret_key -> something like iuyr8934y8yuiroew892iop1oiwos9q3i94o16md, for example
  * AWS_REGION=your_elasticsearch_region -> something like us-east-1, for example

Just a quickly explanation about the connection code above: if you are on local and gonna use a local Elasticsearch instance - and you just was straight forward on Elasticsearch instalation, defining none security policies - you do not need anything but the Elasticsearch url, that as said, could be something like http://localhost:9200, having a different port depending on port your local Elasticsearch service is running, obviously. For that reason, the first block of if has just the URL - Elasticsearch client needs you put the curly brackets. Otherwise, Rails app will be on production and try to make the access to your AWS Elasticsearch instance.